Skip to content

Deepseek 4: -sm tensor - #26490

Open
am17an wants to merge 2 commits into
ggml-org:masterfrom
am17an:dsv4-sm-tensor
Open

Deepseek 4: -sm tensor#26490
am17an wants to merge 2 commits into
ggml-org:masterfrom
am17an:dsv4-sm-tensor

Conversation

@am17an

@am17an am17an commented Aug 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overview

Add -sm tensor for DeepSeek4. The FA is mirrored because there is only 1 K head (Q has 64). Apart from that add memset_tensor is added to the meta backend.

Additional information

on 4x 4090s I see a PP increase of 50% but TG does not increase.

Requirements

@github-actions github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning labels Aug 3, 2026
@am17an am17an changed the title DSV4: sm tensor Deepseek 4: -sm tensor Aug 3, 2026
@am17an
am17an requested a review from gaugarg-nv August 3, 2026 04:28
@am17an

am17an commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

@gaugarg-nv since Johannes is busy would you mind reviewing this PR?

gaugarg-nv
gaugarg-nv reviewed Aug 3, 2026
View reviewed changes

@gaugarg-nv gaugarg-nv left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you test this with other GPU counts like 2 and 3? Also test with uneven tesnor-split values like {0.7,0.3}? This often helps uncover bugs related to tensor splitting.

Comment thread tests/test-llama-archs.cpp
Comment on lines -427 to -429
if (arch == LLM_ARCH_DEEPSEEK4) {
return false;
}

@gaugarg-nv gaugarg-nv Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WebGPU backend seem to be failing with this change in CI. can you check?

@am17an am17an Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WebGPU can't handle any change in test-llama-archs so I would just ignore this for now.

@am17an

am17an commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Could you test this with other GPU counts like 2 and 3? Also test with uneven tesnor-split values like {0.7,0.3}? This often helps uncover bugs related to tensor splitting.

Thanks, that indeed turn up a bug which is now fixed with 3bbafa2

@iSevenDays

iSevenDays commented Aug 3, 2026

Copy link
Copy Markdown

Is this change compatible with draft model?

E llama_model_load: error loading model hyperparameters: key not found in model: dflash.attention.sliding_window_pattern
E failed to load draft model, 'DeepseekV4-Flash-20260731-DSpark.gguf'

/root/llama.cpp/build/bin/llama-server \
  --model DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
  -ngl 99 -ngld 99 \
  --main-gpu 1 \
  -ot 'blk\.([5-9]|[12][0-9]|3[0-7])\.ffn_(gate|up|down)_exps\.=CPU' \
  --split-mode tensor -ts 43,43 \
  -b 8192 -ub 8192 -fit off -fa on -c 262144 \
  --host 0.0.0.0 --port 8002 --parallel 1 --jinja --reasoning on \
  --no-mmap --log-verbosity 4 \
  --spec-type draft-dspark --spec-draft-n-max 5 \
  --model-draft DeepseekV4-Flash-20260731-DSpark.gguf \
  --spec-draft-p-min 0.5

main model loads fine under --split-mode tensor, but the DSpark draft fails

the same command works on current master (build 10248 / e8e06f7)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gaugarg-nv gaugarg-nv gaugarg-nv left review comments
@JohannesGaessler JohannesGaessler Awaiting requested review from JohannesGaessler JohannesGaessler is a code owner
@CISC CISC Awaiting requested review from CISC CISC is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning testing Everything test related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@am17an @iSevenDays @gaugarg-nv